[FLINK-37586][udf] Add support for options in user-defined functions and update related documentation#4252
Merged
Merged
Conversation
…and update related documentation
github-actions
Bot
added
docs
Member
Author
|
cc @yuxiqian |
yuxiqian
reviewed
Feb 26, 2026
yuxiqian
reviewed
Mar 4, 2026
Member
yuxiqian
left a comment
yuxiqian
left a comment
There was a problem hiding this comment.
LGTM overall.
+@lvyanquan as it is related to public API change.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for passing configuration options to user-defined functions (UDFs) in Flink CDC pipeline definitions. Previously, UDFs could be registered but had no way to receive external configuration at startup.
Changes:
- Extended
UdfDefclass with anoptionsfield, new constructor, and updatedequals/hashCode/toString - Updated the YAML parser to read the new
optionsblock for UDFs, and fixedTransformTranslatorto pass these options through (previously it always passed an emptyHashMap) - Added example UDF implementations (Java and Scala), new test YAML fixture and unit/integration tests, and documentation in both English and Chinese
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
flink-cdc-composer/src/main/java/.../definition/UdfDef.java |
Adds options map field with constructor overloads and updated equals/hashCode/toString |
flink-cdc-composer/src/main/java/.../translator/TransformTranslator.java |
Passes actual UDF options instead of an empty HashMap |
flink-cdc-cli/src/main/java/.../parser/YamlPipelineDefinitionParser.java |
Parses optional options block from YAML UDF definitions |
flink-cdc-cli/src/test/java/.../YamlPipelineDefinitionParserTest.java |
Adds test for parsing UDF definitions with options |
flink-cdc-cli/src/test/resources/definitions/pipeline-definition-with-udf-options.yaml |
Test YAML fixture with options block for a UDF |
flink-cdc-composer/src/test/java/.../FlinkPipelineUdfITCase.java |
Integration test for configurable UDFs using options |
flink-cdc-pipeline-udf-examples/.../java/ConfigurableFunctionClass.java |
Java example UDF that reads options via ConfigOptions |
flink-cdc-pipeline-udf-examples/.../scala/ConfigurableFunctionClass.scala |
Scala example UDF that reads options via ConfigOptions |
flink-cdc-e2e-tests/.../rules/unexpected.yaml |
Updates the allowed-keys error message to include options |
docs/content/docs/core-concept/transform.md |
English documentation for the new options feature |
docs/content.zh/docs/core-concept/transform.md |
Chinese documentation for the new options feature |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
yuxiqian
approved these changes
Mar 5, 2026
ThorneANN
pushed a commit
to ThorneANN/flink-cdc
that referenced
this pull request
Mar 6, 2026
…and update related documentation (apache#4252)
suhwan-cheon
pushed a commit
to suhwan-cheon/flink-cdc
that referenced
this pull request
Mar 9, 2026
…and update related documentation (apache#4252)
ThorneANN
pushed a commit
to ThorneANN/flink-cdc
that referenced
this pull request
Mar 19, 2026
…and update related documentation (apache#4252)
Mrart
pushed a commit
to Mrart/flink-cdc
that referenced
this pull request
Mar 26, 2026
…and update related documentation (apache#4252)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This close https://issues.apache.org/jira/browse/FLINK-37586
This pull request adds support for passing configuration options to user-defined functions (UDFs) in pipeline definitions. It updates both the YAML schema and the internal Java representations to allow specifying an
optionsblock for each UDF, making these options available at runtime. The documentation is updated to explain how to use this feature, and new tests are added to ensure correct parsing and handling.